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Abstract 

AAAS is collaborating with BSCS in the development of a curriculum unit for eighth grade 
students that connects fundamental chemistry and biology concepts to better prepare them for 
high school biology. Recognizing that teachers play an influential role in delivering the 
curriculum to students, we developed teacher support materials and professional development 
designed to help teachers use the unit effectively. In order to leam about the promise of the 
teacher support materials, we developed an assessment targeting aspects of participating 
teachers’ (n = 8) science content knowledge and pedagogical content knowledge (PCK) for the 
specific learning goals of the unit. Specifically, the assessment targeted three areas of teachers’ 
knowledge: 1) content knowledge 2) knowledge of student thinking, and 3) knowledge of 
strategies to move student thinking forward, across four item contexts: 1) chemical reactions, 2) 
conservation of mass, 3) flow of matter in living systems, and 4) plant growth. Teachers took the 
assessment three times: before PD, after PD, and after teaching the unit. The assessment items 
were mainly constructed response and were scored using indicators of success and difficulty. 
Teachers made gains over time in most of the knowledge areas and across most of the contexts. 
Areas where they did not make clear progress, or where their knowledge was particularly low, 
indicated that either the assessment instrument or the teacher support materials could be 
improved. Revisions based on these findings are reported. 


1 



Flanagan et al. NARST 2013 


Introduction 

The Toward High School Biology (THSB) curriculum unit is a replacement unit for the eighth 
grade that is intended to connect key chemistry and biology concepts in order to better prepare 
students for high school biology. A collaborative effort between BSCS and AAAS, the unit is 
being developed in response to the increasingly chemical nature of modern biology (NRC, 

2003), and a need for coherent materials addressing matter transformation and biological growth, 
areas that students have particular trouble with (DeBoer et al., 2009). For more on the learning 
goals of the unit, see Roseman et al. 2013. 

The unit is currently its third year (Year 3) of development. In Year 1 we conducted a small pilot 
test of a draft of the unit with two teachers (Herrmann-Abell, Flanagan, & Roseman, 2012). In 
Year 2 we revised the materials and tested them with eight teachers from four states. For Year 3 
we revised the materials again and are presently conducting a feasibility test that includes a 
small-scale cluster randomized controlled trial in two school districts. 

Teachers play an integral role in how curriculum materials reach students (Ball & Cohen, 1996). 
When teachers have knowledge of student thinking about particular science concepts they can 
respond more productively to students during class (Borko et al. 1992). Participating teachers 
lacked prior experience with many of the activities, pedagogical strategies, and science concepts 
specific to the unit. With this in mind, we developed teacher support materials and professional 
development that were designed to be educative for teachers by directly addressing their needs as 
learners (Ball & Cohen, 1996; Davis & Krajcik, 2005). 

The Teacher’s Edition (TE), which could be accessed online, followed the recommendations in 
the literature (Ball & Cohen, 1996; Schneider & Krajcik, 2002) by including sections on 
background knowledge about the relevant science ideas that went beyond what was expected of 
students, common student misconceptions and naive ideas, the rationale for the choice of the 
phenomena, and chapter and lesson conceptual overviews. The professional development (PD) 
consisted of a three-day face-to-face workshop focused on developing teachers’ knowledge of 
the unit’s content storyline and pedagogical strategies that anticipate and respond to student 
thinking. Teachers were also required to complete lesson analysis tasks throughout the process of 
teaching the unit; the analysis involved interpreting selected written responses from the students’ 
THSB notebooks to assess the conceptual progress of the class. For more on the development of 
the teacher support materials, see Kruse et al. 2013. 

Because teachers enact curricula according to their own understandings and beliefs, which often 
differ between teachers and from the developers’ intentions (Ball & Cohen, 1996), we needed a 
measure of teacher knowledge - both content knowledge and pedagogical content knowledge 
(PCK) (Shulman, 1986) - that could ultimately help us to interpret possible differences in 
student learning gains between classrooms using the THSB unit. This measure would initially be 
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formative, helping us to assess the strengths, weaknesses, and overall promise of the teacher 
support materials in helping teachers implement the materials effectively and with fidelity. 

Measuring teacher knowledge has proven challenging (Rowan, Schilling, Ball, & Atkins- 
Bumett, 2001). Few direct measures of teacher knowledge exist; most studies have relied on self- 
report or proxies such as degrees and certifications (Bucher, 2009). Notable exceptions are the 
ATLAST assessments developed by Horizon Research Inc. (Smith, 2010), the MOSART 
assessments (Sadler et al., 2007), and the DTAMS (Saderholm, Brown, & Collins, 1997). Smith 
(2010) initially attempted to measure three domains of teacher knowledge: 1) science content 
knowledge, 2) knowledge of student thinking, and 3) knowledge of strategies to move students’ 
thinking forward. However, the items were multiple-choice and were unable to fully probe the 
latter two domains. The MOSART items can be used with teachers or students and probe content 
knowledge and misconceptions, but do not directly address knowledge of student thinking or 
pedagogical strategies. The DTAMS likewise focused only on content knowledge and no items 
were developed targeting flow of matter in living systems. No existing instruments could provide 
us with the precision needed for our project as none were aligned to the specific content and 
strategies in the THSB unit. 

This paper reports on the development and initial findings of a measure of selected aspects of 
teacher content knowledge and pedagogical content knowledge (PCK) relevant to the unit’s 
learning goals. Unlike our measure of student understanding (Herrmann-Abell, Flanagan, & 
Roseman, 2012; Herrmann-Abell, Flanagan, & Roseman, 2013), which was informed by many 
years of prior work by our own team (DeBoer et al., 2008) and others, our teacher knowledge 
measure represents an initial attempt. As such, we will report on preliminary findings from Year 
2 about participating teachers’ knowledge, but also challenges we faced with interpreting the 
results and subsequent revisions to the instrument. 


Methods 

Knowledge targeted. In order to learn about the promise of the THSB teacher support materials 
and identify potential strengths and weaknesses, we developed an assessment targeting aspects of 
participating teachers’ science content knowledge and pedagogical content knowledge (PCK) 
related to the specific learning goals of the unit. We developed a framework that articulated the 
knowledge and skills that teachers would need to have in order to use the unit effectively. 
Teachers who are able to use the unit effectively can use the materials to help students 
understand and apply the targeted science learning goals. This framework was based on previous 
research on the knowledge needed for teaching (Ball & Bass, 2000; Shulman, 1986) and it 
guided our decisions about the types of knowledge to target in our assessment. There were two 
broad, overlapping categories: content coherence and pedagogical support. For each of these, we 
outlined what teachers should know (knowledge) and what they should be able to do (skills). For 


3 



Flanagan et al. NARST 2013 


the written teacher assessment, we focused on a subset (in bold below in Table 1) of the 
necessary teacher knowledge: 


Table 1: Excerpt from the Teacher Knowledge and Skills Framework (knowledge only). The 
written teacher knowledge assessment targeted the components in bold. 


Content Coherence 
What teachers should know: 

Pedagogical Support 
What teachers should know: 

• Knowledge of the key ideas about 
matter that are targeted in the unit 

(including their boundaries) and why 
the treatment of energy ideas is not 
included 

• Knowledge of commonly held student 
ideas and how they might be manifest 
in student explanations of 
phenomena 

• Knowledge of the phenomena the 
unit uses to illustrate the ideas 
targeted or to illustrate the 
explanatory power of the ideas and 

why these particular phenomena were 
selected 

• Knowledge of the 
representations/models included in 
the unit, why they were selected, and 
how they are expected to support 
student reasoning about and 
explanations of phenomena 

• Knowledge of the science content story 
line for the unit, what each lesson 
contributes to it, and where students 
are expected to be after each lesson 

• Knowledge of student misconceptions 
documented in the learning research 
literature and how they may be 
manifest in student explanations or 

questions 

• Knowledge of productive student ideas 
that are limited in scope and how the 
unit strives to build on and extend 
them 

• Knowledge of the features provided by 

the student and teacher materials to 
help teachers find out students' ideas 
and help them move towards a more 
scientifically correct understanding 


These bold statements fall under three domains of teacher knowledge: 

1) Knowledge of science content 

2) Knowledge of student thinking about the science content (primarily common student 
misconceptions or naive ideas about the content) 

3) Knowledge of strategies to move student thinking forward (primarily phenomena and 
representations that would provide evidence to challenge students’ misconceptions and 
support the correct science idea) 
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Our assessment was closely linked to a specific curriculum unit and teacher support materials. 
As such, it was designed to measure teacher knowledge through the lens of the specific 
chemistry and biochemistry learning goals of the unit, which include ideas about chemical 
reactions, conservation of mass, and plant and animal growth. For more on the THSB unit 
learning goals, see Roseman et al. 2013. 

Four contexts aligned to the unit learning goals were targeted: 

1) Chemical reactions in the case of polymer formation 

2) Conservation of mass in a sealed container 

3) Flow of matter (a carbon atom) through multiple organisms 

4) Applying conservation to photosynthesis and plant growth 


Assessment design. The assessment was designed to be taken online using a web browser. 
Teachers were instructed to take the test individually, and without referring to any THSB or 
outside resources. The test took about 40 minutes to complete. 

Teachers took the assessment three times: 1) before receiving PD (pre-test), 2) after receiving 
three days of face-to-face PD (post-PD-test), and 3) after teaching the six- week unit (post-unit- 
test). These three time points were chosen to enable us to tease apart the effect of PD and the 
effect of teaching with support materials. There were two versions of the assessment; the pre-test 
and post-unit-test were exactly the same, but the post-PD-test had very similar items that differed 
slightly in their example phenomena. For example, in the post-PD-test the formation of the 
polyester polymer was used instead of the silly putty polymer (PDMS), which was used in the 
pre- and post-unit-tests; the rest of the item structure was exactly the same. This difference in test 
form was intended to reduce tedious repetition for the test-takers, as the post-PD-test occurred 
very soon after the pre-test. 

Each version of the test comprised four multi-part items (Table 1). Most items were constructed 
response, which included providing short answers, lists, and explanations, and a few were 
multiple choice. Two items were aligned to chemistry ideas (chemical reactions and conservation 
of mass), and two items were aligned to biochemistry ideas (plant growth and flow of matter in 
living organisms). Each of the four items had a section targeting each of the three teacher 
knowledge domains: science content, student thinking, and strategies to move student thinking 
forward. 
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Table 1: Each item targeted three areas of teacher knowledge for one of four contexts aligned to the 
science learning goals of the THSB unit. 



Item 1 

Item 2 

Item 3 

Item 4 

Chemical 
reactions in the 
case of polymer 
formation 

Conservation of 
mass in a sealed 
container 

Flow of matter 
(a carbon atom) 
through multiple 
organisms 

Applying 
conservation to 
photosynthesis 
and plant growth 

Knowledge of 
science content 

Constructed 

response 

Constructed 

response 

Constructed 
response; providing 
a sequence of 
chemical reactions 

Multiple choice 

Knowledge of 
student thinking 

Constructed 

response 

Constructed 

response 

Constructed 

response 

Multiple choice + 
explanation 

Knowledge of 
strategies to 
move student 
thinking forward 

Constructed 

response 

Constructed 

response 

Constructed 

response 

Constructed 

response 


Like Smith (2010), we felt that teachers would respond better to taking an assessment that was 
tailored to their profession, as opposed to taking a student assessment to measure their content 
knowledge. An advantage of probing all three domains in our assessment was that by including 
questions about student thinking and pedagogical strategies alongside the content knowledge 
questions, it was apparent to the test-taker that the test was teacher-specific. 

Participants. The participants were eight eighth-grade science teachers from four different states 
(Table 2). All teachers volunteered to participate in the project and test the unit in their 
classrooms. Two of the teachers had used the pilot version of the unit in Year 1 of the project, 
while the other six teachers used the unit for the first time in Year 2. All but one teacher 
completed all three assessments; this teacher did not complete the post-PD-test due to scheduling 
issues. 

Table 2: Information on participating teachers. An "x" in the Y1 column indicates that the teacher had 
taught a pilot version of the unit in Year 1. 


Teacher 

Location 

Sex 

Class Taught 

Y1 

# tests 
completed 

1 

Boston, MA 

F 

8 th grade General Science 


3 

2 

Washington, DC 

F 

8 th grade Physical Science 


2 

3 

Howard County, MD 

F 

8 th grade Physical Science 


3 

4 

Howard County, MD 

F 

8 th grade Physical Science 

X 

3 

5 

Howard County, MD 

F 

8 th grade Physical Science 


3 

6 

Fountain, CO 

M 

8 th grade Physical Science 


3 

7 

Fountain, CO 

M 

8 th grade Physical Science 


3 

8 

Fountain, CO 

F 

8 th grade Physical Science 

X 

3 
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Analysis. The research team generated an ideal response for each item section. The ideal 
response was used to develop indicators of success (correct ideas and appropriate examples) and 
indicators of difficulty (incorrect ideas, inappropriate examples, or missing information). Two 
researchers matched each teacher response to these indicators. Percentages of teachers presenting 
an indicator of success or an indicator of difficulty are reported; half-points were occasionally 
used when answers were ambiguous. For the pre-test and post-unit-test, n = 8. For the post-PD- 
test, n = 7. An example of an item section with ideal response and indicators of success and 
difficulty is provided below. 


Example part of an item targeting content knowledge 
in the context of conservation of mass in a sealed container. 

Students in an 8th grade class are about to study the Toward High School Biology unit. Two weeks 
before they begin the unit, the teacher attempts to find out what students' initial ideas are. The teacher 
places a piece of bread in a plastic bag and seals the bag so that nothing could get in or get out. Then a 
student weighs the bag and its contents. The teacher asks the students to record this weight as “Initial 
Weight." They put bag in a dark place and leave it there for two weeks. After two weeks, students notice 
mold has grown on the bread. 



Before having the students determine the final weight of the bag and its contents, the teacher asks 
them to predict how the final weight would compare to the initial weight, and to give reasons for their 
predictions. 

A. Will the final weight be the same as, more than, or less than the initial weight? Explain why. 

Ideal Response: 

The final weight will be the same as the initial weight. Weight is a measure of the amount of matter. 

The bag is sealed so no matter can get in or out, therefore the mass will not change because the amount 
of matter in the bag does not change. Matter is made up of atoms and atoms cannot be created or 
destroyed, therefore the atoms that make up the mold now must have been in the bag as part of 
something else at the start [because no atoms were allowed to enter or leave the bag]. 
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Indicators of Success: 


Teachers use these ideas: 

1. The final weight will be the same as the initial weight. 

2. Weight is a measure of the amount of matter. (Either explicit or implied) 

3. No substances entered or left the system. 

4. The amount of matter in the bag does not change. 

5. No atoms entered or left the system. 

6. The number of each type of atom in the bag does not change. 

7. As new substances form the amount of the starting substances decrease. (No mention of 
atoms) 

8. New substances (mold) have formed but the atoms that make up these substances were 
inside the bag at the start as part of some other substance. 


Indicators of Difficulty: 


Teachers use these ideas: 

1 . 

The final weight will be more than the initial weight. 

2. 

The final weight will be less than the initial weight. 

3. 

The final weight will be the same as the initial weight because (gives wrong reasoning). For 
example, the mold doesn't weigh anything. 

4. 

Matter was created when the mold grew so the mass will increase. 

5. 

The number of atoms increased when the mold grew so the mass will increase. 

6. 

Matter/atoms entered the system so the mass increased (Missed that the bag was sealed 
but might have an understanding of conservation.) 

7. 

The weight decreases because the mold ate the bread as it grew and during this process 
matter or atoms were destroyed. (Food (the bread) was used for energy only.) 

8. 

Some matter is being changed into energy when the mold grows but that energy cannot 
leave the sealed bag so the weight stays the same. 

9. 

No mention of atoms or molecules. 

10. 

Answers as a student so not sure of what they think the correct answer is. 
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Results & Discussion 


Knowledge of science content. 

Chemical reactions in the case of polymer formation. Teachers were shown the molecular 
structures and properties of the starting and ending substances involved in the formation a 
polymer and asked if a chemical reaction occurred. At all three time points all teachers (100%) 
knew that a chemical reaction had occurred. In their explanations of why a chemical reaction 
occurred, only 50% of teachers included the word “atoms” in the pre-test (and 43% in the post- 
PD-test), but by the post-unit-test, all teachers (100%) mentioned atoms. It was somewhat 
surprising that half of teachers made no mention of atoms in the pre-test, given that the atomic- 
molecular structures were provided in the stem. 

Conservation of mass in a sealed container. Teachers were shown a change occurring in a 
sealed container and asked if the final weight would be the same as, more than, or less than the 
initial weight. The number of teachers providing the correct response (that the final weight will 
be the same as the initial weight) increased following PD (75% pre-test, 100% post-PD-test, and 
100% post-unit test). Unlike the chemical reaction context, atoms were not explicitly mentioned 
or shown in the item stem. While none of the teachers used atomic explanations to account for 
the substance-level phenomenon in the pre-test, they improved over time, especially after 
teaching the unit (0% pre-test, 14% post-PD-test, and 50% post-unit test). For example, Teacher 
4 knew the weight would be the same at all three time points, but her explanation progressed 
from simply stating that the container was closed, to saying that matter could not enter or leave, 
to specifying that atoms or molecules could not enter or leave (Table 3). 

Table 3: Change over time in Teacher 4's responses to an item describing a change occurring within a 
closed container and asking if the final weight would be more, less, or the same as the initial weight. 


Teacher 4 

Response 

Reasoning 

Pre-test 

"The final weight should be the same as the initial weight, 
as it was a "closed system" (the bag was closed)." 

closed system 

Post-PD-test 

"The final mass should be the same, as it is a closed 
system, and no matter has been able to leave or enter the 
jar." 

closed system; 
matter cannot enter 
or leave 

Post-unit-test 

"The final weight should be the same because in a sealed 
container (system) no atoms/molecules can enter or leave, 
so the mass will remain constant." 

closed system; 
atoms/molecules 
cannot enter or leave 
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Flow of matter through multiple organisms. Teachers were told that a carbon atom was taken 
in by a plant, which was then eaten by an animal. This animal was then eaten by another animal 
and the carbon atom was now part of the second animal’s muscle. They were asked to list the 
sequence of chemical reactions that would explain the journey of this carbon atom. Chemical 
formulas and balanced equations were not required - only the names of the reactants and 
products. For each reaction, they were asked to specify which product contained the carbon atom 
and in which organism the reaction was occurring. The percentage of teachers who were able to 
list all of the necessary reactions was very low but increased slightly over time (0% pre-test, 14% 
post-PD-test, and 25% post-unit-test). In all three assessments, all teachers were able to list that 
the carbon atom was in the air as part of carbon dioxide before it was taken in by the plant. The 
percentage of teachers listing a step in which glucose reacts with a source of nitrogen to form 
amino acids increased over time (0% pre-test, 7% post-PD-test, and 50% post-unit-test). 
Similarly, the percentage of teachers ending with the carbon atom as part of a protein molecule 
increased over time (13% pre-test, 43% post-PD-test, and 75% post-unit test). There was also a 
marked decrease in problematic misconceptions following PD. The number of teachers listing 
cellular respiration as a step went from 50% in the pre-test to 0% in the post-PD- and post-unit- 
tests. Similarly, the number of teachers listing energy or the sun as a reactant or product went 
from 50% in the pre-test to 0% in the post-PD- and post-unit-tests. 

Applying conservation to photosynthesis and plant growth. Teachers were given a distractor- 
driven multiple-choice item (Sadler 1998) that asked: after photosynthesis occurs, how will the 
total mass of the ending substances compare to the total mass of the starting substances? On the 
pre-test, 81% of teachers answered correctly, selecting the answer choice indicating that the mass 
would be the same because atoms are conserved. One teacher’s response received half-credit 
because he answered incorrectly but “as a student,” misinterpreting the directions for the item. 
One teacher received no credit because she selected an answer choice indicating that the ending 
substances would have more mass because glucose adds mass to the plant. This distractor 
confuses the increase in the measured mass of the plant when it grows with the total mass of 
products equaling the total mass of reactants. In both the post-PD-test and the post-unit test all 
teachers (100%) chose the correct answer choice. 

Summary and discussion. For three of the contexts, most teachers were able to provide or select 
the correct answer on the pre-test and all teachers were able to on the post-unit-test. However, 
the percentage of teachers who were able to provide a correct sequence of word equations to 
trace the journey of a carbon atom through three organisms was extremely low. This finding was 
concerning, as the task was nearly identical to a task that students are expected to be able to 
successfully complete at the end of the unit. While most of the teachers had biology degrees, 
most of their teaching experience was in physical science. Several teachers also shared with us 
that they had never learned the particular biochemical content knowledge targeted in the THSB 
unit. This is unsurprising as traditional K-12 and college science curricula do not make explicit 
links between the physical and life sciences and polymer formation is not typically treated in 
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lessons on matter and energy in living systems (see Roseman et al. 2013 and Kruse et al. 2013 
for more on the selection of the learning goals and phenomena in the THSB unit). While few 
teachers were able to list the full sequence correctly, even by the end of teaching the unit, it was 
encouraging that after receiving PD and teaching the unit fewer teachers were including 
misconceptions and more teachers were listing more of the necessary steps. 

Interestingly, the two teachers who were able to list the correct sequence of reactions by the end 
of the unit were the two teachers who had co-taught a pilot version of the unit with members of 
the research team the previous year. These two teachers were also among the four teachers who 
got through most of the lessons; many teachers did not get to experience teaching the later 
lessons, which focused on the biochemistry learning goals (see Roseman et al. 2013 for more on 
the Year 2 implementation). While they were not able to provide complete correct answers on 
their pre-tests, these two teachers had a higher level of familiarity with the science content that 
seems to have become fully realized through the process of teaching the unit on their own. The 
other six teachers were learning new science content for the first time during professional 
development, shortly before teaching the unit. 

For the two chemistry contexts, which required teachers to construct explanations, we noted that 
while most or all teachers were able to provide the correct answer, their explanations were 
weaker. However, in both cases they improved over time in their ability to use atomic-level 
explanations for phenomena. 


Knowledge of student thinking. 

Chemical reactions in the case of polymer formation. When asked what misconceptions or 
naive ideas students might have that would influence their ability to recognize whether or not a 
chemical reaction has occurred, on the pre-test only 50% of teachers could list at least one 
misconception. This increased slightly to 57% on the post-PD-test and 63% on the post-unit-test. 
Moreover, for all three time points, no teachers listed more than one misconception (though 
sometimes different teachers listed different misconceptions; they were not all listing the same 
misconception). 

Conservation of mass in a sealed container. There was little change in teacher knowledge of 
student misconceptions for this context. In all three assessments, all teachers (100%) were able to 
list at least one misconception that students might have that would cause them to say the final 
weight would be more or less than the initial weight. However, most teachers only listed one 
misconception, and none listed more than two at any time point. 
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Flow of matter through multiple organisms. Again, there was little change in teachers’ 
knowledge of student misconceptions. On all three assessments, all teachers (100%) were able to 
list at least one misconception. However, few teachers listed more than one or two. 

Applying conservation to photosynthesis and plant growth. For this context, teachers were 
given a large multiple-choice item with nine misconception-based distractors (the same item that 
was used to probe their content knowledge in this context). They were asked to explain what 
they think the most popular incorrect answer choice(s) would be for students prior to instruction 
and why. At all three time points, all teachers (100%) were able to select at least one incorrect 
answer and explain what ideas students who select it might hold. However, only a few teachers 
listed more than one incorrect answer and underlying idea. For this particular item, this may have 
been partly due to the design of the item. Forthcoming revisions to the items and instrument will 
be discussed later. 

Summary and discussion. For three of the item contexts, all teachers were able to list at least 
one relevant student misconception or naive idea at all three time points, meaning they came to 
the project with some knowledge of student thinking in these areas. However, teachers were less 
knowledgeable about student thinking in the context of chemical reactions, and only a few 
progressed over time. In contrast to the science education research literature, which reports many 
common misconceptions for each of the science concepts targeted by the items (e.g. Anderson et 
al., 1990; Marmaroti et al. 2006), most teachers were only able to list one misconception per item 
context. A recent study of elementary teaches found that many teachers do not think that students 
have prior ideas about science and are “blank slates.” Though these teachers had heard of the 
work done on students’ science misconceptions and were shown examples, a third of the teachers 
could not provide a single example of a student misconception (Gomez-Zwiep, 2008). It may be 
that while our teachers knew about research on misconceptions, they found it hard to link these 
to the specific content and their experience with real student responses. Teachers completed 
lesson analysis tasks throughout their implementation of the unit that required interpretation of 
student thinking through their written work. In theory, these tasks could have helped to develop 
their knowledge of student thinking. However, teachers completed these tasks very literally, 
without attending to underlying student thinking, and they tended to complete them long after 
completing the lessons they were associated with, eliminating the potential for timely responses 
to student difficulties. 
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Knowledge of strategies to move student thinking forward. 

Chemical reactions in the case of polymer formation. When asked about what evidence they 
could use to help students with misconceptions about chemical reactions, there was little 
consistent change in teachers’ responses over time. On the pre-test only 38% of teachers could 
provide at least one source of evidence that would be useful in moving student thinking forward. 
On the post-PD-test this was 43% and on the post-unit-test, 25%. This could be partially 
explained by the teachers’ lack of knowledge of student misconceptions in this content area. If 
they are unaware of student thinking, it is logical to conclude that they also would not have 
knowledge of strategies for moving student thinking forward. There was also an issue with the 
structure of the item that could have placed limitations on teachers’ answers: teachers were only 
asked to provide activities for misconceptions they had listed for this context. This will be 
discussed in a section on revisions to the instrument. 

Conservation of mass in a sealed container. Teachers were asked how they would help 
students who made incorrect predictions about the weight reconcile their prediction with their 
observations. A major theme in the THSB unit is the use of models to explain phenomena. The 
ideal response would include the use of molecular models to make concrete the link between 
atom conservation and mass conservation. The number of teachers suggesting using molecular 
models was initially low but increased by the post-unit-test (13% pre-test, 29% post-PD-test, 
50% post-unit-test). Three of the four teachers who suggested using molecular models in the 
post-unit-test also mentioned atoms in their content knowledge explanations of why the weight 
would be the same in the post-unit-test. This could hint that teachers who were thinking about 
the content on the atomic level for themselves may have been more able to see how molecular 
models could be helpful for students. 

Flow of matter through multiple organisms. Teachers were asked about what evidence they 
would show students to help students with misconceptions. There was notable progress in this 
area as teachers moved from didactic, teacher-centered suggestions toward presenting students 
with evidence and allowing students to grapple with and construct the science ideas themselves. 
The number of teachers who cited at least one way to present evidence that contradicts common 
misconceptions increased over time (25% pre-test, 72% post-PD-test, and 88% post-unit test), 
while the number of teachers writing that they would “tell, teach, remind, review” the correct 
ideas with students in a didactic manner decreased (75% pre-test, 43% post-PD-test, and 31% 
post-unit-test). 

Applying conservation to photosynthesis and plant growth. When asked what evidence they 
would use to help students who respond to the photosynthesis item incorrectly, on the pre-test 
63% of teachers provided at least one relevant phenomenon or modeling activity. This increased 
to 71% on the post-PD-test and 81% on the post-unit test. However, few teachers listed more 
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than one activity; again, this could be due to the structure of the item which only asked about 
misconceptions they had selected for this context. 

Summary and discussion. A recent review reported that there have been few studies focused on 
teachers’ thinking about engaging students with science phenomena; their ideas about linking 
challenging science concepts with the phenomena and guiding students to develop scientific 
explanations are largely unknown (Schneider & Plasman, 2011). In this project we have started 
to probe this domain of PCK in the context of specific curriculum materials. For three of the item 
contexts, teachers made gains over time in their ability to provide appropriate phenomena and 
modeling activities that could serve as evidence to contradict misconceptions and move student 
thinking forward. For the chemical reactions context, knowledge was lower and no consistent 
progress was made; however, this could be partially explained by the item design. Answers 
improved most dramatically for the flow of matter context and change from didactic strategies 
toward student-centered strategies was clearly visible. In a study comparing the PCK of novice 
teachers who were content experts in photosynthesis and novice teachers who were content 
novices, the content experts were more aware of students’ conceptual problems. However, both 
groups lacked knowledge of “suitable experiments or demonstrations” for helping move student 
thinking forward for this content (Kapyla, Heikkinen, & Asunta, 2009). The implication is that 
this aspect of PCK must be explicitly taught, separate from content. The improvement we saw in 
this domain suggests that the PD and support materials were successful in helping teachers to 
build this knowledge. 


Conclusions and Forthcoming Revisions 

Teacher content knowledge. For three of the four contexts, most teachers could supply correct 
answers to items probing their content knowledge; those who could not in the pre-test typically 
could by the post-unit-test. However, for the context of flow of matter through multiple 
organisms, teachers’ biochemical content knowledge was very low. Even by the post-unit-test, 
no teacher who was teaching the unit for the first time in Year 2 was able to provide a correct 
sequence of reactions, a task that students were expected to be able to do upon completion of the 
unit. However, individual features of their responses indicated that while they were not 
producing complete correct answers, their answers were improving and getting closer to the ideal 
response. While teachers were generally able to supply correct answers (with the exception of 
the flow of matter context), their explanations were typically briefer than the ideal response and 
much less organized and precise. We have made revisions to the assessment, detailed below, to 
see if teachers’ explanations improve with a clearer prompt. However, it is also possible that 
because the teaching profession is primarily oral, our teachers had little recent practice with 
written explanations. 
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Teacher PCK. Teachers had a basic awareness of student misconceptions and could usually list 
one or two per context. However, they made no obvious improvement over time, and their 
knowledge of chemical reactions misconceptions was especially low. In contrast, teachers made 
obvious gains in their knowledge of activities that could provide students with evidence to 
counteract their misconceptions. The exception was the chemical reactions context, for which 
their knowledge of misconceptions was lowest. The gains teachers made in their content 
knowledge and knowledge of strategies for moving student thinking forward indicate that the PD 
and educative teacher support materials show promise. 

Revisions to the instrument. We have recently revised the assessment for our Year 3 field test 
this spring in response to our findings from Year 2. While the Year 2 data were very informative, 
the process of rating the responses also helped us to uncover some ambiguity in the items that 
could be eliminated and thereby increase our confidence in the assessment’s ability to accurately 
measure teacher knowledge. 

The brevity of the responses we received from teachers - in contrast to the fairly lengthy ideal 
responses generated by the research team - indicated that for some items, directions for how to 
respond to the items and what level of detail was expected may have been inadequate. As a 
result, teachers may not have been clear about the importance of being thorough, precise, and 
exhaustive in their answers. In addition teachers may have felt like they were writing to people 
they knew personally (the research team) and were therefore writing more informally than they 
would if they were modeling an answer for students. In the Year 3 assessment, for constructed- 
response items targeting content knowledge, we added the direction: “Be sure to include a 
complete explanation and write as if you were modeling a correct response for your students.” 
Additionally, a few teachers were occasionally confused about whether a question targeting their 
own content knowledge was supposed to be answered “as a teacher” or “as a student.” In the 
Year 3 assessment, we prefaced items targeting teachers’ own content knowledge with: “What 
would be the ideal correct response to the question: ...” 

We removed multiple-choice entirely from the assessment. When we provided teachers with a 
large multiple -choice item with many misconception-based distractors and asked them to choose 
the one(s) they thought would be most popular with students, we removed the need for teachers 
to think of misconceptions on their own. Based on the data from the constructed-response items 
targeting knowledge of student thinking, teachers only came up with one or two per content area, 
so the multiple-choice format was obscuring an area where teachers were likely to have 
difficulties. While we wanted teachers to choose all of the distractors for which they could 
provide good explanations of underlying student thinking for, the act of selecting from given 
choices probably undermined this and caused teaches to be selective and choose only one or two. 
Furthermore, probing content knowledge with multiple choice items was less informative than 
with constructed response items because most teachers we able to select or give a right answer, 
but they often struggled to write good explanations for it. Multiple choice items gave us no 
information about teachers’ abilities to develop explanations about the science content. 
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Noting that teachers often only listed one misconception in items targeting their knowledge of 
student thinking, we wanted to be sure that this accurately represented the extent of their 
knowledge and not concision, test-fatigue, or misinterpretation of the item. In the Year 3 
assessment, for constructed-response items targeting student thinking, we added the direction: 
“Include as many as you can.” 

In items targeting knowledge of strategies to move student thinking forward, teachers were asked 
to supply strategies for each of the misconceptions they had listed. This task was therefore 
limited by teachers’ narrow knowledge of student misconceptions. For the Year 3 assessment, 
we simply asked teachers to “include all that you think could be useful.” Additionally, teachers 
often simply provided a list of activities without making explicit links to how each activity 
would be helpful. While we had hoped that teachers would spontaneously make these links, we 
recognized that our items may not have clearly prompted this level of detail. For the Year 3 
assessment, we added scaffolding in the form of a flexible table. Column headings for 
“Activity/Demonstration/Data Set,” “Intended observation,” and “How it contradicts incorrect 
student idea(s)” were provided and rows could be added or deleted freely (Figure 1). 



Figure 1: Screen capture from the Year 3 online teacher knowledge assessment interface showing the 
scaffolded table targeting teachers' knowledge of strategies to move student thinking forward. 


Revisions to the TE and PD. Since the Year 2 implementation of the unit, much work has been 
done to add content to and improve the usability of the teacher support materials. A brief 
overview of these revisions is provided here; for more detail, see Kruse et al. 2013. 

In Year 2 the Teacher’s Edition was accessible online but there was no print edition. This meant 
that teachers could not easily (and did not, based on observations and classroom video) refer to it 
on the spot during class. Even outside of class, a complicated menu system made it difficult to 
quickly find specific documents and information. For the Year 3 field test, all teachers will have 
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a print TE with facing student and teacher pages. We look forward to seeing if this increase in 
usability results in 1) increased use of the teacher support materials and 2) larger increases in 
teachers’ content knowledge and PCK across the three assessment time points. 

The Year 3 student and teacher materials place much more explicit emphasis on the practice of 
developing explanations - linking claim, evidence, and science ideas - than the Year 2 materials. 
This year we will see if guided experience with learning (in PD) and teaching (assisted by the 
TE) this practice will positively impact the characteristics of teachers’ constructed explanations 
of science content on the assessment. 

In response to teachers’ lack of progress in their knowledge of student thinking, the Year 3 
lesson analysis tasks were revised. In the new iteration, teachers are required to complete the 
tasks immediately after the associated lesson. They are also given a finer rating scale to 
accommodate incomplete or ambiguous statements, and are given explicit instructions to pay 
attention to student ideas more than specific words. 

One-shot professional development workshops are known to have little effectiveness in changing 
teaching practice (Lumpe, 2007). While time constraints in both Year 2 and Year 3 only allowed 
for three days of face-to-face PD, in Year 3 we have opted to space out the first two days of PD 
and the last day of PD and assign “homework” in between. We have also provided new tutorials 
and how-to videos that teachers can access online at any time. Additionally, there will be a 
synchronous webinar held during implementation. These features will help create an ongoing PD 
experience. 

Next steps. Ultimately we hope to use a version of this measure of teacher knowledge in a larger 
scale efficacy study of the THSB unit. Collecting information on teacher knowledge - especially 
when we are no longer working with volunteer teachers - could help explain possible differences 
in student performance between classrooms. A question will be if the open response format, 
which provides rich information about teachers’ PCK and their skill in the practice of developing 
explanations, will remain practical and cost-effective. 
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